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Using Nominal Group Technique to Develop a Consensus Derived 
Model for Peer Review of Teaching Across a Multi-school Faculty 


This paper describes the development of a peer review of teaching model for the Faculty of Health at the 
University of Newcastle, Australia. The process involved using the nominal group technique to engage 
Faculty academic staff to consider seven key decision points that informed the development of the peer 
review of teaching model. Use of the consensus based nominal group technique established collegial 
discussion and networking among participants, and has expanded the discussion within the Faculty about 
peer review of teaching. Although the academics come from a diverse multi-school and multi-program 
Faculty, there was high levels of consensus on the structure and process of the developed peer review of 
teaching model. 


group technique, peer review, health professional 


Tracy Burrows, Naomi Findlay, Chloe Killen, Shane E. Dempsey, Sharyn Hunter, Pauline Chiarelli, and 
Suzanne Snodgrass 


This journal article is available in Journal of University Teaching & Learning Practice: http://ro.uow.edu.au/jutlp/vol8/iss278 




Burrows et al.: Development of a Model for Peer Review of Teaching 


Introduction 

As applied to the University setting, peer review of teaching (PRoT) involves academic 
colleagues giving and receiving guided critiques on one another’s teaching practices. PRoT 
can be used to provide feedback about the effectiveness of various teaching approaches in 
promoting student learning (Harris et al. 2008), and it is recognised as a valuable supporting 
tool for academics in higher-education settings to evaluate and develop their teaching 
(Simpson 1995; Harris 2005; Philip & Wozniak 2009). 

While the PRoT process is often described in an international setting (Lomas & Nicholls 
2005; Kell & Annetts 2009; McMahon et al. 2010), a search of the literature and university 
web sites by the authors found that it was not a widely reported professional activity in 
Australian universities. Previous efforts to undertake PRoT in the Australian tertiary sector 
have generally been conducted using informal methods in small-scale projects, which often 
means the processes and results are not publicly documented or available for use (Harris et al. 
2008). 

Currently, the majority of teaching and learning evaluations conducted in Australian 
universities take the form of student feedback (Barries & Ginns 2007; Tucker et al. 2008; 
Australian Universities Quality Agency 2007). Methods such as these, while a valuable tool in 
providing general feedback, are often limited in accurately gauging the quality of teaching and 
learning, due to low completion rates by students, respondent bias and a lack of specificity to 
individual teaching and learning approaches and institutions (Douglas & Douglas 2006). 
PRoT, however, can provide a more universal approach to the evaluation of teaching. 
Designed to accommodate the wide spectrum of university teaching and learning situations 
and contexts, PRoT extends beyond classroom teaching and presentation to include design, 
preparation, learning environment and student engagement. 

In recent years, there has been a developing awareness and recognition of the importance of 
PRoT in Australia (Harris et al. 2008). The Australian Learning and Teaching Council 
(ALTC) recognises PRoT as an integral and necessary part of quality teaching, and has created 
a handbook to support the development of PRoT within Australian higher-education 
institutions. The handbook outlines four situations, each with a specific design guide, where 
PRoT can be implemented: programs to enhance the teaching environment; programs to raise 
the standard of teaching; programs specifically for new staff; and programs specifically for 
sessional staff (Harris et al. 2008, pp. 26-59). 

The handbook recommends for consideration seven key decision points essential to the 
development of a PRoT framework that enhances the learning environment (Harris et al. 2008, 
pp. 32-37): 

1. Whose teaching will be reviewed? 

2. What will be the policy regarding participation? 

3. What will be reviewed? 

4. Who will the reviewers be? 

5. What form will the review process take? 

6. What reporting will take place? 

7. What type of follow-up will occur after completion of the peer review process? 

The Faculty of Health at the University of Newcastle comprises a diverse group of full-time, 
part-time, conjoint and clinical academics and health professionals, teaching in a range of 
undergraduate and postgraduate programs. The Faculty is structured around four schools, each 
of which comprises multiple academic or health professional disciplines, staff and programs. 
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The professional-entry programs of the Faculty include three- and four-year undergraduate 
degrees, and graduate-entry masters' programs. Programs are delivered in a variety of ways 
including lectures, tutorials, labs, problem-based learning sessions, workshops, exhibitions, 
web sites and e-learning. Professional placement and experiential learning form a significant 
part of the teaching and learning style within the various professional-entry programs. The 
faculty staff also contribute to coursework postgraduate studies and research higher-degree 
teaching and learning. Thus there are a wide variety of teaching and learning contexts and 
situations within the Faculty, making it important to embrace evaluation processes that are 
sufficiently rich and informative across this diversity. 

The University of Newcastle has developed two sets of tools students use when reviewing the 
quality of teaching and courses: Student Feedback on Teaching and Student Feedback on 
Courses. However, there are currently no tools available within the University to help staff 
undertake peer review of teaching. 

The aim of this project was to develop an engaging, consensus-building process that would 
result in the development of a framework for peer review of teaching in the Faculty of Health 
at the University of Newcastle. Suggested outcomes from this research process included: 

1. Increasing the accessibility of peer review of teaching for Faculty academic staff by 
encouraging and engaging academic staff to assist in the development of a PRoT 
framework and tools. 

2. Providing a framework, process and resources specifically designed to assist Faculty 
academic staff with peer review of teaching. 

3. Enhancing the teaching and learning environment for staff and students 

The methodology for the research-based project involved three stages: a literature review to 
identify existing PRoT frameworks and tools; five consensus-building forums of staff and 
students using Nominal Group Technique (NGT) to identify the requirements of a PRoT 
framework for the Faculty; and the research team's development of the framework and tools 
based on the outcomes of the previous stages. This paper describes Stage 2: the nominal group 
technique process used within the study and the outcomes of the five consensus building NGT 
sessions. 

Methods 

Ethics 

Approval for this quality-assurance teaching activity was obtained from The University of 
Newcastle Human Research Ethics Committee. 

Research Team 

The research team was established in March 2009 and comprised academic teaching staff from 
each School within the Faculty of Health. A research assistant (RA), who was a beginning 
academic staff member of another Faculty, was appointed to assist the team. However, as the 
research developed the RA became an active and integral member of the research team. The 
team consisted of two academic staff from Radiation Therapy, two from Physiotherapy, one 
from Nutrition and Dietetics, one from Nursing, one from Pharmacy and one from 
Communication and Media in the Faculty of Science and Information Technology (the RA). 
The team included males and females, as well as a mix of new and experienced staff. This 
blend of staff allowed for a diversity of teaching and learning situations to be considered when 
developing the PRoT process. 

Consensus Building Process 

The Nominal Group Technique (NGT) was chosen as the consensus-building process to be 
used to gather and prioritise information from academic staff members. The NGT method was 
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used because it is an inclusive process designed to encourage all participants to contribute 
equally and democratically (Delbecq & Van de Ven 1971; Center for Disease Control and 
Prevention 2006; Chappie & Murphy 1996), and it has been used successfully in a range of 
other consensus-building exercises within both the academic and health environments (Vella 
et al. 2000; Bamford & Warder 2001; Jones 2004). 

The six steps in the NGT process proposed by Jones (2004 p. 22) are: 

1. Individual generation of ideas 

2. Recording of all participants’ ideas (in a round-robin format) 

3. Group discussion of all generated ideas (to organise the list and remove duplications) 

4. Preliminary vote to select the most important ideas 

5. Group discussion of the vote outcomes (including additions and further merging of 
overlaps) 

6. Final voting on the priority of items 

The active engagement of all participants during the NGT process meant that the outcomes of 
the NGTs were not subject to interpretation by the facilitator, nor dominated by the more 
vocal group members. An important benefit of the NGT process for this project was that it 
allowed for a consensus on the day of the NGT (Bamford & Warder 2001, p. 318) rather than 
taking extra analysis time (Jones 2004, p. 23). Given the diverse nature of teaching practices 
across the Faculty of Health, the NGT process also supported collegial relationships among 
academics when discussing teaching and learning in a wide variety of contexts. 

In the NGT process in this project, each nominal group session ran for approximately 120 
minutes and consisted of between seven and 10 participants; this number has been previously 
identified as being ideal for conducting NGT sessions (O'Neil & Jackson, 1983 p. 132). Each 
session was conducted by a research team member and audited for consistency by the RA. 
Within each nominal group session, participants were asked to respond to three of the decision 
points proposed by the ALTC handbook: 

1. What will be reviewed? 

2. Who will the reviewers be? 

3. What form will the review process take? 

The research-team member facilitating the nominal group session outlined the process in 
detail, addressing any questions, and introduced the questions to the participant group one 
question at a time in order. Each question was displayed on a Powerpoint slide, and all 
participants were instructed to anonymously provide up to five responses to each question by 
writing on index cards that were subsequently collected by the RA. All responses were then 
transcribed by the RA onto a poster, which was displayed for the entire group to view. The 
research team member led open discussion on each item to ensure that all group members 
understood the item, and where an idea was suggested multiple times or where the group 
recognised highly similar items, these items were arranged with direction from the participants 
to form a single item. Participants then independently ranked the items on the revised list into 
their top three choices on a small card; the RA collected the cards and collated the results. The 
resulting ranked order of items represented the group's prioritised preferred responses to each 
question. The results were then fed back to the group. This process was followed for each 
question. 

Next, participants were asked to partake in an open group discussion of the remaining decision 
points: 

1. Whose teaching will be reviewed? 
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2. What will be the policy regarding participation? 

3. What reporting will take place? 

4. What type of follow-up will occur after completion of the peer review process? 

The RA took notes to reflect the range of discussion on these questions, taking care to 
preserve the participants’ anonymity. Upon completion of the nominal group sessions, the 
project team met and compiled the results from all groups to develop an initial consensus. This 
was tabulated and emailed to all the participants of the nominal group sessions. All 
participants were invited to provide feedback on the draft consensus document by attending a 
follow-up workshop or via email. The results of the workshop informed the development of 
the final consensus document for peer review of teaching in the Faculty of Health at the 
University of Newcastle. 

NGT Participants 

To ensure that the voices of staff from all schools were heard, the team proposed to run a 
nominal group session for each school within the Faculty (four schools), and one at the 
Ourimbah campus (Central Coast, NSW). All full-time and part-time academics from the 
Faculty of Health (n=189 FTE) were invited to participate in the consensus-building exercise 
via email invitation. To ensure all staff received an email invitation, initial email invitations 
were sent by Heads of Schools, and by the Research & Research Training Officer of the 
Faculty. A second follow-up invitation email was sent approximately one week later from a 
member of the research team. Participants were allocated to one of four nominal group 
sessions: Health Science; Nursing and Midwifery; Biomedical Science and Pharmacy; and 
Ourimbah Campus (mixed discipline). Staff of the School of Medicine and Public Health did 
not participate at this stage of the research. However, several staff who participated from the 
other schools taught in Medicine and Public Health, so they were represented as much as 
possible. 

Students were invited to contribute to the research, and academics approached student 
program committees seeking interest and involvement from members. Written invitations 
were provided to interested students, and a fifth nominal group session was undertaken to 
accommodate students who volunteered. 

Results 

A total of 29 academics and four students participated across the five nominal groups (Health 
Science = 7, Nursing and Midwifery =10, Biomedical Science and Pharmacy = 6, Ourimbah 
= 6). Results of the top three priorities identified from each nominal group for questions 1 to 3 
are presented below, as are the results of the open group discussion for questions 4 to 7. 

Q1: What will be reviewed? 

This question generated the most variation in responses, with 13 different suggestions 
received, ranging from teaching methods, assessment items and online coordination to clinical 
facilitation. Although there were differences in the rankings and how they described their 
answers, academics and students consistently prioritised lectures, tutorials and laboratories as 
priorities for review. These activities represent for most academics the most common teaching 
and learning contexts in which they engage with students, and participants felt that as a 
starting point for being reviewed, this was what they required feedback on in the first instance. 

Three categories of items to be reviewed emerged from their responses: 

1. Topic-content descriptors: up-to-date knowledge and skills, content structured 
correctly, quality of information 

2. Delivery and communication descriptors: clarity of message, communication skills, 
student engagement and interaction, materials used, lesson plan 
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3. Learning-outcomes descriptors: facilitation of learning, checking of learning, learning 
benefits, outcomes of planned activities. 

Q2: Who will the reviewers be? 

Responses to this question were the most consistent, with only five responses generated from 
all NGT groups. To provide for review of topic content and teaching and learning style and 
outcomes, academics widely agreed that reviewers should: 

• be staff trained in the peer-review process who can use the designated framework 

• include content experts or specialists who understand the program and professional 
context being presented 

• include education experts who can review and advise on matters related to student 
learning or teaching delivery. 

Q3: What form will the review process take? 

Seven suggestions were provided from NGT groups; these varied from timing of reviews 
(once per semester or year, or biannually) through to the process being compulsory or 
mandatory especially for new staff and the review being performed in reciprocal pairs. 
Academics felt that the PRoT process should use standardised tools, in which both reviewers 
and reviewees would be trained, to facilitate appropriate feedback. Reviewees should be able 
to contact trained reviewers with whom they felt comfortable to review them. The reviewee 
and the reviewer should be able to meet prior to the session to discuss the session's objectives, 
how it fits into their program, and how it is to be delivered. It was felt that this process would 
facilitate a non-threatening, relaxed and less-formal review. 

Q4: Whose teaching will be reviewed? 

Discussions points from each group revealed a general consensus that all people involved in 
teaching - including experienced and new academics, sessional staff, clinical facilitators and 
even guest lecturers - should have the opportunity to have their teaching reviewed if desired. 

Q5: What will be the policy regarding participation? 

There was majority consensus that participation should be voluntary. However, the student 
NGT group interestingly suggested that the policy should be mandatory as a matter of quality 
assurance to ensure that students are receiving the best education and learning environment 
possible. The majority of participants agreed that peer review of teaching might be linked to 
staff performance measures, probation requirements and mentoring of academic staff, in 
addition to be being used on a personal level for inclusion in individuals’ teaching portfolios. 

Q6: What reporting will take place? 

All groups agreed that reporting should initially be between the reviewee and the reviewers. 
Most groups agreed that reporting post-review should take place face-to-face, and all agreed 
on the need for a formal written document to underpin the face-to-face discussion. All written 
reporting should be in an official form with both parties maintaining professionalism during 
the process and being aware of the possibility that the reports could be used for other purposes 
such as academic promotion. There was also the suggestion that reports should be delivered in 
unalterable format. 

Some participants felt that the results should be sent to additional people such as the Head of 
School or a mentor to ensure accountability of teaching, but there was not a general consensus 
on this. 

Q7: What type of follow-up will occur after completion of the peer review 
process? 
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Most groups felt it was important for the process to be followed up at a later stage to give 
reviewees the chance to get feedback on whether they had met their objectives for 
development as identified in the first review. 

Discussion 

While the NGT has been used in a range of teaching research settings (Bamford & Warder 
2001; Philip & Wozniak 2009; Jones 2004), and there is some data using the NGT to assess 
students’ evaluation of teaching and education experiences (Chappie & Murphy 1996; 
Williams et al. 2006), there is little specific research using the NGT in a whole-of-Faculty 
approach to the development of a PRoT process. This study used the NGT as a means of 
gaining consensus from a large and diverse group of academic staff about the PRoT process. 
The strong consultation and democratic methods used within the nominal group process 
allowed Faculty staff to take ownership of the development of a PRoT framework and tool. 
This is one of the identified strengths of using this type of consensus-built process in the 
context of PRoT (Harris et al. 2008). 

Results collected during this project through the NGT method are similar to those collected 
previously by survey design (Cosser 1998; Kohut et al. 2007). The current study identified 
that despite the broad array of different undergraduate and graduate entry courses and 
programs, and teaching and learning activities that exist within the Faculty of Health, there 
were commonalties in regard to academics' views and opinions on key decision points about 
peer review. 

Participants in this study felt that reviewers involved in providing feedback on an individual’s 
teaching need to be trained in the peer-review process. These results are similar to previous 
research findings that identify the importance of feedback and evaluation of teaching being 
approached ethically and with sensitivity and respect, such that the review process is non¬ 
threatening and conducted in collegial environments (Handal 1999; Bell 2005). Similar to the 
results obtained by the NGT process, Kohut et al. (2007, pp. 21-22) found in a large, survey- 
based study with 223 academics across a range of Faculties, that respondents wanted 
reviewers for PRoT to complete training about the process. Participants from the Kohut et al. 
study (2007, pp. 22-23) suggested offering training for observers, and that reviews should 
follow professional ethical guidelines and involve some sort of classroom or lecture 
evaluation. The participants in this research had similar suggestions. 

Participants in this study suggested that PRoT should be considered at all levels of teaching 
experience, not only for new staff but for those who are experienced. This finding is 
significant given that PRoT is typically associated with new staff undertaking a graduate 
certificate or tertiary teaching qualification. In a recent ALTC review of PRoT processes in 
Australia universities, it was found that limited peer review occurs in tertiary settings; in those 
where PRoT is undertaken, it is often only completed for new staff or as part of the teaching 
curriculum for a graduate certificate in tertiary teaching (Harris et al. 2008). The importance of 
PRoT in a tertiary setting has also been acknowledged by its prioritisation internationally as 
means for evaluating and improving teaching in higher education (Irby 1983; Morehead & 
Shedd 1997). A similar study by Cosser (1998 p. 158), using an open-ended qualitative 
survey, identified that the reviewer should be someone who understands the subject area: an 
experienced teacher or one possessing formal education training. The respondents to the 
survey identified the need for a standardised tool that includes both quantitative and 
qualitative components. In addition, Cosser’s study revealed that when asked "What aspects of 
teaching should be reviewed?" (a question similar to one in the current study), the results of 
the top four responses included presentation of material, material, communication skills and 
presentation style (Cosser 1998, p. 147). Tutorials were also identified as an important 
category. This demonstrates a consistency in responses across studies for academic staff. 
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The current study obtained information about peer review from a diverse range of academics 
in higher education from a variety of health backgrounds. A study by Snell et al. (2000, p. 2) 
suggests that teachers in medical and health fields with a clinical component should be 
evaluated in a broader spectrum, and that this evaluation should include all domains relevant 
to their teaching objectives, extending beyond classroom teaching to “knowledge, scholarship, 
clinical competence, teaching effectiveness and professional attributes”. These additional 
components, while not identified as priorities in this study, demonstrate the importance of 
obtaining evaluation and feedback on teaching from a range of sources. In the current study, 
participants reported that the information and documentation gathered during PRoT would be 
highly valuable in both in terms of quality assurance and for use when striving toward 
excellence in teaching and promotion. This was suggested by groups wanting the PRoT 
process to be linked to performance indicators, and by students wanting the PRoT to be 
mandatory to ensure they were receiving the best quality of education. The development of the 
PRoT has generated a team of academics within the Faculty that have knowledge and skills to 
undertake the consensus-built process throughout the university in a variety of settings. The 
process and outcomes have been showcased to academic staff across the university in teaching 
and learning workshops and to a national collective of higher-education health staff with an 
interest in the outcomes and process. It is expected that this process will be piloted and 
undertaken in various Faculties across the University of Newcastle within the next few years. 

The current study is limited in that the number of suggestions shown here does not reflect the 
total output from NGT groups; rather, the results reflect the groups' compacted lists of the 
items as voted on by participants, and this should be considered when interpreting findings. 
The NGT method may not always be appropriate for collection of information from larger 
groups of people, where surveys or focus groups would be better suited. For future studies, the 
length of time to conduct an NGT could be shortened with trained facilitators and by including 
a slightly larger sample in each nominal group. Practical consideration for implementation of 
the NGT method in other settings would include: training in NGT, staff to facilitate them, 
appropriate resources and incentives to encourage staff to participate. 

Conclusion 

Within this research, the use of the nominal group technique allowed academics in the Faculty 
of Health to reach consensus on the structure and process to be adopted in the development of 
a PRoT framework. Using a collaborative technique to engage academics in discussion has 
helped establish a greater collegial network and the growth of a culture of peer review of 
teaching as an integral part of teaching and learning review in the Faculty. The process 
highlighted the importance of academic peers within the review process; as a result, the 
consensus among participants was to use the phrase ‘Peer-Assisted Review of Teaching’ 
(PARoT) to reflect the collaborative nature of the process. 

Since the completion of this research, a Peer-Assisted Review of Teaching (PARoT) 
framework and tool have been developed within the Faculty of Health. The framework and 
tool were piloted during both teaching semesters of 2010, by a small number of academics 
(n=30) across various schools within the Faculty. Further collaborative meetings were held to 
discuss the process and outcomes of the use of the tools, and the outcomes of being reviewed 
and being a reviewer. Funding from the University’s Centre for Teaching and Learning has 
supported this research and will assist in making the final consensus-agreed documents 
available as an e-resource to the Faculty staff in 2011. Additionally, a larger research project, 
which attempts to extend and include the PARoT NGT process to all Faculties of the 
University of Newcastle, is in development. 
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